Vector and Matrix Calculus

• For a morphism \(f\colon X\to Y\), the gradient \(\nabla f\colon X\to Z\) is a linear map, such that \[ dy = \langle \nabla f, dx\rangle \] in which bilinear map \(\langle \cdot, \cdot \rangle\colon Z\times X \to Y\) is well-defined.

• This can also be written in terms of differential form as: \[ dy = df(dx). \]

• Del in cylindrical and spherical coordinates - Wikipedia

• Divergence of a vector field \(\mathbf{F}\) is \[ \nabla\cdot \mathbf{F} = \frac{\partial F_{x_i}}{\partial {x_i}}. \]

1. The net flux through a unit volume.
2. The rate of change of the ratio of volume (the ratio of the rate of change in volume, rate of change of a unit volume) subjected to the flow of a vector field.
   • For a vector field given by a linear transformation:
     • \[ \nabla\cdot(\mathbf{Ax}) = \frac{d}{dt}\ln V \]
   • The infinitesimal transformation generated by the vector field \(\mathbf{F}\) is: \[ \tilde{x}^i = x^i + F^idt \]
     • The Jacobian of the transformation would be: \[ J_i^j = \begin{bmatrix} 1 + \partial_{x^1}F^1dt & \partial_{x^2}F^1dt & \cdots & \partial_{x^n}F^1dt \\ \partial_{x^1}F^2dt & 1 + \partial_{x^2}F^2dt & \cdots & \partial_{x^n}F^1dt \\ \vdots & \vdots & \ddots & \vdots \\ \partial_{x^1}F^ndt & \partial_{x^2}F^ndt & \cdots & 1+ \partial_{x^n}F^ndt \\ \end{bmatrix} \]
     • And the determinant is: \[ \det J_i^j = 1 + \nabla\cdot \mathbf{F}\,dt + O(dt^2) \]
     • By taking the derivative of that: \[ \frac{d}{dt} \det J_i^j = \nabla\cdot \mathbf{F} \]

• \[ \nabla^{\cdot 2} f = \nabla\cdot\nabla f \]
• \(\nabla^2\) is used in physics, and \(\Delta\) is used in mathematics.

• Divergence of Gradient
• Trace of the Hessian .

A Jacobian matrix of a vector field \(\mathbf{f}\) is \[ J^{i}{}_{j}=\frac{\partial f^i}{\partial x^j} \] where \(i\) is the row number and \(j\) is the column number.

It tells the rate of change in the vector field in any direction. Consider the identity: \( \mathrm{d}f^i=J_{j}^{i}\mathrm{d}x^j \) or equivalently, \( \mathrm{d}\mathbf{f}=\mathbf{J}\mathrm{d}\mathbf{x} \).

Beware that some people prefer to use the transpose of this Jacobian as their Jacobian.

\[ J^{-1}{}^i{}_j := \frac{\partial x^i}{\partial f^j} \] The inverse matrix can also be written concisely as \[ J^{-1}{}^i{}_j = J_j{}^i. \]

• Reciprocate each element and transpose the Jacobian matrix.

A Jacobian of coordinate transformation from coordinates \(x^j\) to coordinates \(\tilde{x}^i\) is \[ J^i{}_j=\frac{\partial \tilde{x}^i}{\partial x^j} \] which transforms the components.

To transform the basis, the inverse Jacobian is used. \[ \frac{\partial}{\partial \tilde{x}^j}=J_j{}^i\frac{\partial}{\partial x^i} \] equivalently, \[ \begin{bmatrix}\tilde{\mathbf{e}}_{1}&\tilde{\mathbf{e}}_{2}&\cdots&\tilde{\mathbf{e}}_{n}\end{bmatrix}=\begin{bmatrix}\mathbf{e}_{1}&\mathbf{e}_{2}&\cdots&\mathbf{e}_{n}\end{bmatrix}\mathbf{J}^{-1}. \]

The determinant of the Jacobian is the ratio of volumes due to transformation. Thus used as the factor in the change of the measure of an integral.

Hessian \(\mathbf{H}\) of a twice-differentiable scalar field \(f\) is: \[ H_{ij} = \frac{\partial^2 f}{\partial x^i\partial x^j}. \]

• Hessian matrix is the transpose of the Jacobian matrix of the gradient.
  • excalidraw:./hessian.excalidraw
  • \((\mathrm{d}\mathbf{x})^{\rm T}\mathbf{H}[f]\mathrm{d}\mathbf{x} = (\mathrm{d}\nabla f)^{\rm T}\mathrm{d}\mathbf{x}.\)
  • If it is evaluated at a stationary point, then \(\mathrm{d}\nabla f\) would point in the direction of the gradient \(\nabla f\).
  • Notice that \(\nabla f\) is the normal map, namely, a Gauss map.
• If the Hessian is positive-definite at \(\mathbf{x}\), then \(f\) attains an isolated local mimimum at \(\mathbf{x}\), by the same note, if the Hessian is negative-definite, then \(f\) attains an isolated local maximum.

\[ \frac{d}{dx}(\mathbf{A}\mathbf{B}) = \frac{d\mathbf{A}}{dx}\mathbf{B} + \mathbf{A}\frac{d\mathbf{B}}{dx} \]

\[\frac{d\mathbf{A}^{-1}}{dx}=-\mathbf{A}^{-1}\frac{d\mathbf{A}}{dx}\mathbf{A}^{-1}\]

• \[ \mathbf{A}\mathbf{B} = \mathbf{B}\mathbf{A} \iff e^{\mathbf{A}}e^\mathbf{B} = e^{\mathbf{A}+\mathbf{B}} \]
• \[ e^\mathbf{O} = \mathbf{I},\quad \left(e^\mathbf{A}\right)^{-1} = e^{-\mathbf{A}},\quad \left(e^\mathbf{A}\right)^n = e^{n\mathbf{A}} \]
• \[ \left(e^{\mathbf{A}}\right)^{\mathrm T} = e^{\mathbf{A}^\mathrm{T}}, \quad \operatorname{det}\left(e^\mathbf{A}\right) = e^{\operatorname{tr}(\mathbf{A})} \]
• If \(\mathbf{A}\) is diagonalizable: \[ e^{\mathbf{A}} = \mathbf{V}e^{\mathbf{\Lambda}}\mathbf{V}^{-1}. \]
• The solution to the differential equation: \[ \mathbf{y}' = \mathbf{A}\mathbf{y} \] is the matrix exponential: \[ e^{\mathbf{A}t}\mathbf{y}_0 \] for any square matrix \(\mathbf{A}\).

• \[ \frac{d}{dt}\det \mathbf{A}(t) = \operatorname{tr}\left(\operatorname{adj}(\mathbf{A}(t))\frac{d\mathbf{A}(t)}{dt}\right) \] where \(\operatorname{adj}\) is the adjugate matrix.
• If \(\mathbf{A}\) is invertible, it can further be said to be \[ \frac{d}{dt}\det\mathbf{A} = \det(\mathbf{A}(t)) \operatorname{tr}\left(\mathbf{A}^{-1}(t)\frac{d}{dt}\mathbf{A}(t)\right) \]

• This means
  • \[ \frac{\partial \det\mathbf{A}}{\partial A_{ij}} = (\operatorname{adj}\mathbf{A})_{ji} = (\mathbf{C})_{ij}, \]
    • where \(\mathbf{C}\) is the cofactor matrix;
  • \[ d\det(\mathbf{A}) = \operatorname{tr}(\operatorname{adj}(\mathbf{A})\,d\mathbf{A}) = \langle (\operatorname{adj}\mathbf{A})^{\rm T}, d\mathbf{A}\rangle_{\rm F}, \]
    • where \(\langle \cdot,\cdot\rangle\) is the Frobenius inner product;
  • \[ \nabla \operatorname{det}(\mathbf{A}) = (\operatorname{adj}\mathbf{A})^{\rm T} = \mathbf{C}, \]
    • where \(\nabla\) is the gradient.